This invention relates to computer-aided synthesis of visual images. More particularly, this invention relates to a system and method for synthesizing image pairs from a single high dynamic range image used to produce a single percept.
An additional image domain is introduced when extending from monocular display to binocular display. Existing binocular display systems only utilize this additional image domain for stereopsis. Human vision is not only able to fuse two displaced images, but also fuse two images with difference in detail, contrast and luminance, up to a certain limit. This phenomenon is known as binocular single vision. Humans can perceive more visual content via binocular fusion than just a linear blending of two views.
The popularity of 3D movies leads to the wide availability of low-cost binocular display devices. While the dual display domains (one for the left eye and the other for the right eye) double the space of visualization, existing binocular displays only serve for one type of binocular vision, stereopsis. Another commonly experienced binocular vision phenomenon in daily life is binocular single vision (or singleness of vision), i.e. images from two eyes are fused and perceived as a single percept, even though these two images are different (FIG. 2) [Howard and Rogers 2002]. Such image fusion is not a simple blending, but a complex non-linear neurophysiological process [MacMillan et al. 2007]. The first two rows of FIG. 2 illustrate the difference between fusion (the third column) and the linear blending (the fourth column). In addition, it tends to retain higher contrast, sharply focused, and brighter content from either view during the single percept formation [Steinman et al. 2000]. In other words, via the dual display, it is feasible to present more human-perceivable visual content than any single image, as vision can naturally combine two images without being aware of the difference between two images.
Unlike binocular display, high-dynamic range (HDR) display is less accessible to the general public. Even though tone mapping can be adapted to present the HDR content on a low-dynamic range (LDR) display, there is a tension between displaying large-scale contrast and fine-scale details. Striking a good balance is always challenging.
Binocular Single Vision
It is known how to combine different images, such as stereopsis [O'Shea 1983], which differs from combining different images from two eyes into a single vision or percept [von Helmholtz 1962]. While binocular single vision occurs only in a small volume of retinal area around where eyes are fixating, stereopsis occurs at places even where eyes are not fixating (when images of object appear double). It was discovered that such a fusion process is a non-linear combination of luminance, contrast, and color. To prove this, MacMillan et al. [2007] measured the interocular brightness response using asymmetrical neutral density filters and Baker et al. [2007] measured the interocular contrast response using sine-wave gratings.
This non-linear fusion is a complicated neurophysiological procedure and is generally regarded as a combination of binocular fusion and suppression [Ono et al. 1977; Steinman et al. 2000]. Binocular fusion is a process of superimposing and combining similar content from the two views into one unified and stable percept, which happens when the two views are similar or identical (FIG. 2, upper row). Binocular suppression occurs when one view (submissive) is blank, significantly dimmer, much more blurry, or has significantly less contrast than the other (dominant). In this case, a single percept is formed in the human vision system by smartly turning off all or part of the submissive view (FIG. 2, middle row). However, when the two views are too different (e.g. FIG. 2, bottom row), an undesirable phenomenon, binocular rivalry, occurs. In this case, the result is a non-converging percept composed of continuously alternating “patches” from the two views [Lei and Schor 1994], as both stimuli are too strong and none of them can suppress the other. Obviously, such continuous alternation can be noticed by an observer and causes viewing discomfort. Besides binocular rivalry, sometimes binocular suppression may also lead to visual discomfort when the stimulus is too strong. A halo or drifting can be observed as a result of inhibitory effect at the center-surround receptive fields excited by the contour [Lei and Schor 1994].
The above discomforts can greatly impede or destroy the visual experience. Hence, what is needed is an assessment tool for binocular viewing comfort. The need for an assessment tool seems to suggest an image similarity metric. There are several existing metrics, including mean squared error (MSE), structural similarity (SSIM) [Wang et al. 2004], perception-oriented metrics Visible Difference Predictor (VDP) [Daly 1993] and its extension High Dynamic Range Visible Difference Predictor (HDR-VDP, HDR-VDP-2) [Mantiuk et al. 2005; Mantiuk et al. 2011]. Known metrics consider the visible difference between two images when the observer looks at these images with both eyes. However, these existing metrics do not consider the binocular vision in which the left eye and right eye of observers are presented with two different images. An obvious shortcoming of existing metrics can be illustrated by binocular suppression (FIG. 2, middle row) where two images are obviously different using any existing metric, even though a stable percept can be formed. Hence, none of the existing metrics can be applied.
Tone Mapping:
Several sophisticated tone mapping techniques have been proposed to generate LDR images from HDR images. Reinhard [2006] provided a comprehensive survey on tone mapping techniques, ranging from sigmoidal compression to image appearance model, and to perception and engineering-based methods. Tone mapping methods can be roughly classified into global and local operators. Histogram adjustment methods and adaptive logarithmic mapping such as [Larson et al. 1997; Drago et al. 2003] are two main categories of global operators. On the other hand, there are also several prevalent local operators, such as bilateral filtering approach [Durand and Dorsey 2002], gradient domain optimization [Fattal et al. 2002] and perceptual-based contrast processing [Mantiuk et al. 2006].
Known tone mapping operators can be used as building blocks for a binocular tone mapping framework to generate two LDR images that optimally increase the human-perceivable visual content without triggering discomfort.
It desirable to produce two views that are as different as possible in order to retain more visual content from the source HDR. However, there is a limit on the difference between two views. When such limit is exceeded, binocular viewing discomfort appears, and even worse, binocular single vision may fail. Such viewing discomfort [Lambooij et al. 2009] is an important health issue receiving much attention due to the wide availability of 3D displays.
The following references provide background to the present invention:    BAKER, D., MEESE, T., MANSOURI, B., AND HESS, R. 2007. Binocular summation of contrast remains intact in strabismic amblyopia. Investigative ophthalmology & visual science 48, 11, 5332.    BLAKE, R., AND BOOTHROYD, K. 1985. The precedence of binocular fusion over binocular rivalry. Attention, Perception, & Psychophysics 37, 2, 114-124.    CARTER, R., AND HUERTAS, R. 2010. Ultra-large color difference and small subtense. Color Research & Application 35, 1, 4-17.    CHEN, H., AND WANG, S. 2004. The use of visible color difference in the quantitative evaluation of color image segmentation. In Acoustics, Speech, and Signal Processing, 2004. Proceedings. (ICASSP '04). IEEE International Conference on, vol. 3, IEEE, iii-593.    DALY, S. 1993. The visible differences predictor: an algorithm for the assessment of image fidelity. Digital images and human vision 11.    DRAGO, F., MYSZKOWSKI, K., ANNEN, T., AND CHIBA, N. 2003. Adaptive logarithmic mapping for displaying high contrast scenes. In Computer Graphics Forum, vol. 22, Wiley Online Library, 419-426.    DURAND, F., AND DORSEY, J. 2002. Fast bilateral filtering for the display of high-dynamic-range images. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques, ACM, 257-266.    EHRENSTEIN, W., ARNOLD-SCHULZ-GAHMEN, B., AND JASCHINSKI, W. 2005. Eye preference within the context of binocular functions. Graefe's Archive for Clinical and Experimental Ophthalmology 243, 9, 926-932.    FATTAL, R., LISCHINSKI, D., AND WERMAN, M. 2002. Gradient domain high dynamic range compression. ACM Transactions on Graphics 21, 3, 249-256.    HOWARD, I., AND ROGERS, B. 2002. Seeing in Depth, vol. 1. I Porteous.    KOOI, F., AND TOET, A. 2004. Visual comfort of binocular and 3D displays. Displays 25, 2-3, 99-108.    LAMBOOIJ, M., IJSSELSTEIJN, W., FORTUIN, M., AND HEYNDERICKX, I. 2009. Visual discomfort and visual fatigue of stereoscopic displays: a review. Journal of Imaging Science and Technology 53, 030201.    LARSON, G., RUSHMEIER, H., AND PIATKO, C. 1997. A visibility matching tone reproduction operator for high dynamic range scenes. Visualization and Computer Graphics, IEEE Transactions on 3, 4, 291-306.    LEI, L., AND SCHOR, C. 1994. The spatial properties of binocular suppression zone. Vision research 34, 7, 937-947.    LEVELT, W. 1965. Binocular brightness averaging and contour information. British journal of psychology 56.    LIN, M., AND JANE, S. 2009. Analysis of color difference in digital proofing based on color management system. Journal of Communication and Computer 6, 10, 32.    LIU, L., TYLER, C., AND SCHOR, C. 1992. Failure of rivalry at low contrast: Evidence of a suprathreshold binocular summation process. Vision Research 32, 8, 1471-1479.    MAC MILLAN, E., GRAY, L., AND HERON, G. 2007. Visual Adaptation to Interocular Brightness Differences Induced by Neutral-Density Filters. Investigative ophthalmology & visual science 48, 2, 935.    MANTIUK, R., DALY, S., MYSZKOWSKI, K., AND SEIDEL, H. 2005. Predicting visible differences in high dynamic range images: model and its calibration. In Proc. SPIE, vol. 5666, SPIE, 204-214.    MANTIUK, R., MYSZKOWSKI, K., AND SEIDEL, H. 2006. A perceptual framework for contrast processing of high dynamic range images. ACM Transactions on Applied Perception (TAP) 3, 3, 286-308.    MANTIUK, R., KIM, K., REMPEL, A., AND HEIDRICH, W. 2011. Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. In ACM Transactions on Graphics (TOG), vol. 30, ACM, 40.    MYSZKOWSKI, K. 1998. The visible differences predictor: Applications to global illumination problems. In Rendering techniques '98: proceedings of the Eurographics Workshop in Vienna, Austria, Jun. 29-Jul. 1, 1998, Springer Verlag Wien, 223.    NORTON, T., CORLISS, D., AND BAILEY, J. 2002. The psychophysical measurement of visual function. Butterworth Heinemann.    ONO, H., ANGUS, R., AND GREGOR, P. 1977. Binocular single vision achieved by fusion and suppression. Attention, Perception, & Psychophysics 21, 6, 513-521. O′SHEA, R. 1983. Does stereopsis have a fusional component? Attention, Perception, & Psychophysics 34, 6, 599-603.    REINHARD, E. 2006. High dynamic range imaging: acquisition, display, and image-based lighting. Morgan Kaufmann.    STEINMAN, S., STEINMAN, B., AND GARZIA, R. 2000. Foundations of binocular vision: A clinical perspective. McGraw-Hill Medical.    TREISMAN, A. 1962. Binocular rivalry and stereoscopic depth perception. The Quarterly Journal of Experimental Psychology 14, 1, 23-37.    VON HELMHOLTZ, H. 1962. Helmholtz's treatise on physiological optics, vol. 3. The Optical Society of America.    WANG, Z., BOVIK, A., SHEIKH, H., AND SIMONCELLI, E. 2004. Image quality assessment: From error visibility to structural similarity. Image Processing, IEEE Transactions on 13, 4, 600-612.    WATSON, A. 1987. The cortex transform: rapid computation of simulated neural images. Computer Vision, Graphics, and Image Processing 39, 3, 311-327.    WOPKING, M. 1995. Viewing comfort with stereoscopic pictures: An experimental study on the subjective effects of disparity magnitude and depth of focus. Journal of the Society for Information Display 3, 3, 101-103.